Entities can include people, vehicles, houses, etc. Entity association in the context of gathering and relating entity data for defense, surveillance systems, sports and entertainment archiving systems is traditionally accomplished using text or structured data, such as known affiliations. In such contexts, it would be desirable to associate structured text data with images and/or video taken of a scene to enhance the meaning of the structured text data and allow for the extraction of meaningful inferences about the data with a high degree of certainty. For example, if a plurality of trucks is traveling together on a highway for an extended period of time, it can be inferred that the collection of trucks are traveling in a convoy. Thus, the video would be tagged with the label “convoy.” In another example, a person is seen entering a car driven by another person. Then, the two persons are likely to know each other.
It would be desirable to associate visual attributes to entities and with video imagery. Persistent and wide-area coverage of video imagery provides an opportunity to monitor the behavior of entities, such as vehicles, people and sites, over long periods of time and large geo-spatial extents. It would also be desirable to deduce the relationship of entities under different contexts and in the presence of clutter and under uncertainties inherent in detecting, classifying and tracking entities from video data. Any entity information derived from videos has an associated probability or belief computed from the data. Inferences of associations use propagation of uncertainties within a network representation built from the data. Therefore, linkages can be established and hidden relationships can be discovered among entities automatically.
Accordingly, what would be desirable, but has not yet been provided, is a system and method for effectively and automatically detecting and relating entities from video data, deducing inferences from the data and their relationships, automatically constructing entity networks, and storing and later retrieving the entity networks for later analysis.